Version History

The following is a list of patch notes for the previous three minor updates.

Patch notes for updates not on this list can be found on the releases page on GitLab.

0.13.x

Note

Despite the jump in versions, 0.13.x is a direct descendant of 0.11.x. It is a migration of the functional 0.11.20 engine onto the 0.12.5 codebase. In theory, this ensures non regression of features and functionality from 0.12, along with some unreleased 0.12.5 fixes.

0.13.11

[Clean]

• Moved from flake9 + Black to Ruff

• Improved type hinting and formatting in several places

[feature]

• Added force argument to fetch_results, which will force runners to accept the remote files and states

• Added header_only to Script/Computer. If this is True, attempts to generate only the resource request header & shebang

• Created a url_helpers module with a conda activation function

[tests]

• BaseTestClass can now reset the URL before cleanup, preventing a timeout if we’re testing a broken host

0.13.10

[performance]

• Runner no longer accesses the Database, instead deferring to Dataset

[feature]

• Destructive actions like wipe_runs now ask for confirmation. (Set confirm=False to ignore this)

[fixes]

• Runner jobscripts no longer require that sourcedir is set, and will default to $PWD, allowing faster debugging

[tests]

• Testing now uses temporary directories for function execution, adding redundancy against file leakage

0.13.9

[features]

• Computer & Script can now be created using template_path, instead of relying on the template path search.

[fixes]

• URL.transport now inherits the raise_errors setting of the parent.

[clean]

• Various updates to code readability

• Remove more instances of _utc tagged timestamp methods

[refactor]

• Deprecate Dataset.run_log, replace with print_run_log

• Update how Dependency handles the internal Dataset list

[tests]

• testingbaseclass renamed to basetestclass

0.13.8

[features]

• Added Runner.last_state_time(state), which will return the most recent timestamp for state

[fixes]

• Dependencies now chain any database updates for any stored Dataset

• Dependencies with differing remote_dir now raise a clear error

• Fixed an issue when attempting to format the output of an iterable within a template

• run_summary_limit is now configurable at Dataset creation

• Internal handling of times is now in line with reality

• Dataset can now set shell and submitter at init

• Version comparison can now handle wildcards

[clean]

• Minor readability fixes to some errors and logs

[refactor]

• BaseTestClass object is now part of the main source, rather than being imported from the testing directory

• Runner.last_submitted now uses last_state_time

• Dataset.run() is now split into stage(), transfer() and run()

• Dataset.run() now always prints a condensed summary

• rsync now uses the --checksum flag by default

• Runner now skips reading local files if it has never been run

[docs]

• Fixed many cases of missing docstrings and type hints

[CI]

• Migrated the Dataset, Dependency, Runner and Database notebook suites

0.13.7

[fixes]

• Fixed an issue related to run_dir and Dependency

[refactor]

• Refactored out the DIR_{UUID} syntax from jobscripts

0.13.6

[ docs]

• Added a page dedicated to changing the Transport module

• Minor cleanup and readability improvements

[feature]

• rsync now checks that the locally installed version is high enough

• Added URL.reset_cmd_history

• Computer.from_yaml can now override settings

• TrackedFile now provides the chmod method, applies the given permissions to the local copy

[fixes]

• URL now resets the cmd history prior to a deepcopy to prevent a crash

• Fixed a bug where multi line functions would be incorrectly stored if they also had inner functions

• Fixed a bug where children would search in an invalid location for parent results

[refactor]

• Transferred internal script files are now made executable by chmod

• Dataset no longer logs when a Database is deleted, or cannot be deleted because it does not exist

[CI]

• Removed the migrated storage and decorator notebook test suites

0.13.5

[features]

• It is now possible to specify (and access) iterables such as dicts and lists in Templates

• Templates can now handle quoted : characters

• Templates now allow escaping characters with ``

[fixes]

• Fixed a bug caused by concatenating ints to a string using _ as the separator

• Fixed an instance where querying Substitution.shortform_op would cause a crash

[refactor]

• Script now recursively links values

0.13.4

[features]

• You can now disable the timeout of a cmd call by setting timeout to 0, False or a negative number

• url.cmd(...) now supports the with context. Calling cmd in this manner ensures that a process is killed in the event of an exception. Useful in combination with asynchronous

• CMD objects can now kill other PIDs, use with care

• URL can now create a tunnel to a host using url.tunnel(local_port, remote_port)

[refactor]

• URL is no longer serialised by Dataset

• Dataset.from_file now requires a url, and will print a warning if not provided

• Unloaded has been removed, objects are now expected to function without placeholders

[fixes]

• URL.ping and URL.test_connection now uses a randomised files and dirs

• URL.gethome() no longer returns None

• ssh_prepend can now be set in URL initialisation

[CI]

• Standard testing suite now uses the logical flag, making use of threading

• The FunctionCache and Connection notebook suites have now been migrated into the standard suite

0.13.3

[refactor]

• Increased scope of replaces and requires in templates

• Computer.pack now uses a more readable format

• Dataset now handles file dumping, rather than relying on SendableMixin

• JUBEInterop module updated to use the Computer module

[feature]

• replaces and requires will now parse a comma separated string into a list of values

• Added Computer.to_yaml and from_yaml which uses the new readable pack

• Computer now also accepts a path to a template file

[tests]

• Computer template functionality is now checked for regression

• Refactored tests that tested BaseComputer to now test Computer

• Added tests for JUBETemplate

• Serialisation tests migrated into standard

• Transport tests migrated into standard

0.13.2

[fixes]

• fixed a bug that could occur when setting extra on Dataset while using a Computer

0.13.1

[refactor]

• remotemanager.logging has been renamed to remotemanager.logging_utils. This should prevent some potential name clashes with the inbuilt logging module

• Templates are now handled by a dedicated Script class

• Deprecate BaseComputer

[feature]

• Dataset now accepts a Script entity as the function argument, generating a script (executed by url.shell) using the arguments as input

• Added a Computer class that handles Templates more elegantly than BaseComputer

• Substitution objects now have a hidden parameter that prevents them from being added to the jobscript

• Substitution objects now have an empty_treatment parameter that dictates how they are treated when empty

• Substitution objects now have a static parameter that forces non-evaluation

[tests]

• The standard test suite is now also tested with Python 3.7

[docs]

• Updated documentation surrounding templates and jobscripts

• Updated documentation surrounding non-function runs

• Moved deprecated documentation to a “legacy” directory. It’s still available, but is untested and will be removed eventually

0.13.0

[features]

• Version can now “imply” the patch (and minor) versions if not provided (defaults to 0)

• add a match functionality to Version, checking against a generic version string like 0.13.x

• Database now warns using more fine-grain version incompatibilities

• Function now has a return_annotation property

[fixes]

• Version now has a rigid output format for properties

• Templates can now properly handle boolean operations (extends to min, max, etc.)

• You can now specify #extra# within templates

• Dependencies no longer uses a legacy run method

• Function.args is now much more reliable

• Function no longer ignores the return type annotation

[refactor]

• Migrated some dependency functionality from 0.12.5

[tests]

• Standard tests now check that randomly generated directories do not already exist before executing

[clean]

• Database write error is now more clear

0.11.20

[fixes]

• Removes breaking issues with the 0.12.0 engine by replacing it with the 0.11.19 engine

[refactor]

• Backport features from 0.12.4 onto the 0.11.19 engine

v0.12.x

Warning

All 0.12.x versions should be considered unstable. This version changes the engine in such a way that is incompatible with some machines. 0.11.20 is a feature backport of these versions. If you experience missing outputs using these versions, you should back up your data and update to 0.11.20.

0.12.4

[feature]

• Added the force keyword to fetch_results(), this will ignore any remote Dataset errors and continue with the fetch

0.12.3

[feature]

• extra_files_send/recv can now be passed via a {local:remote} dict pair. This allows for fine control of where files will be transferred

[fix]

• TrackedFile objects passed to extra files no longer have their dirs ignored

• Dataset.run_path now defaults to remote_dir if nonexistent

• ensure_list no longer drops the values of a passed dict, instead encasing it within a list

[refactor]

• Runner and Dataset now handle extra files via the ExtraFilesMixin class

• TrackedFile objects are now represented via "{file}" shorthand

• rsync no longer wastes a command on creating a local dir

[tests]

• Added testing sanitisation for file paths

• pytest-xdist testing now uses a standardised testing class

0.12.2

[fix]

• Dataset.copy_runners now creates deep copies of runners

• Fixed some instances of Function incorrectly serialising a function

[refactor]

• DynamicMixin now also validates on value access

• Copied Runner objects now update their name

• RunnerState is now less eager to claim that it is in a failed state

• Database now waits for temporary file until a timeout, rather than a flat delay

• ensure_list now uses a try/catch methodology

• ensure_filetype now returns the input string if target type is None

• get_version is now lazy

[docs]

• Fix missing clarification regarding use on Windows

[CI]

• Notebook setup now installs [dev] extras

• Use pytest-xdist in standard test suite

0.12.1

[refactor]

• Dataset.run_args is now a modifiable dictionary containing the globally set parameters and defaults

• Runner.run_args is now a modifiable dictionary containing the Runner level _overrides_ (Empty by default)

• The run_args used by a runner are now available at Runner.derived_run_args

[fixes]

• Fixed a crash involving ensure_list and integers

[docs]

• Improved the docs page on run_args

0.12.0

[refactor]

• The manifest file is now the absolute source of truth for Runners. It contains status updates, stdout and stderr. All remote run behaviour is now based on the contents of this file

• Error files no longer exist, reducing the amount of files present

• Dataset.wait will raise an exception if an error is encountered in a dependency situation. This behaviour differs slightly from a non dependent situation, where wait simply exits

• Runner job submission commands are now generated as a list by Runner.generate_runline()

• check_runner_outputs has been removed

• Internal timestamps are now in UTC format

• Dependencies use more Dataset and Runner code for staging

• Function now uses regex to extract the name and signature from string inputs

[fixes]

• The initial run_cmd is no longer always asynchronous

• Runner no longer accepts state changes from before the current run

• Fixed several instances of Function not properly inserting *args and **kwargs

• Fixed Function discrepancies between python object and string based initialisation

[features]

• Added Runner.stdout and Runner.stderr

  • You can now use the print() function within your jobs

  • Access to manifest.write() still remains

• TrackedFile now has a local_time_utc property, which calls os.path.getmtime() and converts to UTC

[tests]

• Moved several tests to the standard directory, to be tested outside of nbval

v0.11.x

0.11.19

[features]

• wait can now await a single runner, pass a Runner object to wait to await only that runner

• BaseComputer now raises an exception when attempting to set a value that already exists

• Substitution objects now return a clearer error when attempting to create a dynamic link

[fixes]

• Generalised more code relating to Resource/Substitution

• Temporary arguments now set their values properly

[refactor]

• Function now has more common code when storing callable or string inputs

[clean]

• The first submission function now has a docstring

0.11.18

[refactor]

• Jobs are now submitted via bash function, instead of a single line command

• Dataset.reset_runs() now defaults to wipe=False

• URL.cmd now returns a CMD object on dry_run

[fix]

• Fixed a crash where RunnerState would error on comparison to a non RunnerState object

• RunnerState <= and >= operators now function correctly

0.11.17

[fixes]

• Fixed an issue where using more than one = symbol in a template would cause a failure

[features]

• DynamicValue objects can now handle boolean operations (<, <=, =>, >, ==, !=)

[refactor]

• Added try_value function, which must now be used internally when comparing DynamicMixin and DynamicValue objects

0.11.16

[feature]

• Runners now use a manifest file for updating their status and checking run times

• Added ds.update_runners(), which updates the runner history with the manifest content

[fixes]

• Runner history is now sorted by time

• Runner history insertion now checks that the state does not already exist in the whole history, rather than just against the current state

• Added finished property to runner states

• Added sub() function to TrackedFile, allowing for inplace substitutions

[docs]

• Added a page detailing the manifest.write function, and loaded

[refactor]

• Repo and Manifest depend on an internal uuid, which can be overridden by dependencies

• Dependencies now use the Dataset._write_to_repo function, rather than implementing their own

[tests]

• Added a CI test that ensures that a Dataset will be able to skip after an update

0.11.15

[features]

• Added the JUBEInterop module. Currently only supports JUBETemplate

• Computers can now accept “semantic” time formats for arguments with format=time. For example: “24h” will be converted to “24:00:00”

• Template based Computers can now use the “wipe” or “local” methods when handling missing arguments

• DynamicValue will now attempt to be more clear when warning about an overwrite

[fixes]

• An error is now raised if templates have multiple substitutions with differing kwargs

• BaseComputer.required now checks all arguments, not just Resource

• Substitution now also checks availablity using regex

• Computer Arguments now accept the json-style “true” and “false”

[refactor]

• BaseComputer file downloading is now done within a staticmethod

• Added a special case for entry_format when format is “time”

0.11.14

[fixes]

• Fixed an issue where Substitution objects would not be applied if their output evaluated to False

0.11.13

[fixes]

• Fixed an issue where Substitution objects would fail to apply to more than just the first instance

0.11.12

[docs]

• Added initial tutorial page on running a Dataset without a function

[features]

• Added Script, available at remotemanager.connection. Subclass of BaseComputer, but provides a run() method.

• Dataset can now take None as a function. Runner arguments will then be passed directly to the script() method of the url, and the stdout captured as the result.

[fixes]

• Setting temporary values in run() no longer permanently updates arguments

• Fixed a bug that caused default values to always be treated as str in evaluations

[refactor]

• Transport is now a property of the URL, rather than Dataset

• URL now has a uuid, based off connection parameters

• BaseComputer now has a uuid, based off URL.uuid and arguments

0.11.11

[features]

• Templates can now accept python f-string style values, creating Dynamic links where possible

• Added computers.concat_basic, allowing concatenation of strings with Dynamics

• Added target_kwargs to Substitution, provides a dict of the originally specified kwargs

[fixes]

• Dynamics now cast to str when attempting to mix strings with non strings

[docs]

• Added docs for dynamic templates

[refactor]

• default property of Dynamic arguments now uses the correct formatting

• Substitution objects are now created using a new from_string classmethod

• Added utils.Tokenizer, which aids in digesting arbitrary code

[performance]

• Optimised the order that DynamicValue calculates operations

[tests]

• Added some non nbval testing

0.11.10

[fixes]

• Enables the optional keyword for substitutions

0.11.9

[features]

• Added stream option to CMD, which will attempt to stream stdout to the terminal

• Added progress option to rsync, which adds the --progress flag to transfers and streams the output

[refactor]

• Added output for running a dataset

• Verbose output for transferring files now a part of Transport

• Verbose properties are now granted through the @make_verbose class decorator

• Added h flag to default rsync flags

[fixes]

• Fixed instances of spurious ) characters being added to logging calls

[CI]

• Update CI scripts for new ubuntu container

0.11.8

[features]

• Resource and Substitution objects can now set (and store) their value

  • Use collect_values=False when storing a BaseComputer to avoid this behaviour

[refactor]

• Updated how verbose works, now decoupled from logging

• Logs are no longer yaml format

• Removed LoggingMixin

• Removed Quiet

• Updated f-string logging calls with preferered lazy method

• Logger now warns only when force deleting a logfile with content

[fixes]

• DynamicMixin no longer checks for only Resource when linking

• Logger now only logs when required, no longer creating an empty file

[tests]

• Replaced nbval cell comments with jupyter cell tags

0.11.7

[fixes]

• Fixed a data loss issue related to yaml.dump

[clean]

• Update license dates

• Update pypi information

[docs]

• Add quickstart info to README

0.11.6

[features]

• Computers now save the current remotemanager version when storing to dict (disable with include_version=False)

• BaseComputer subclasses that fail to call the super().__init__ now raise a more helpful error when generating a script

[docs]

• Minor clarity improvements

• Add some missing docstrings and typehints

[refactor]

• Common code of Resource and Substitution moved to their mixin class

• RemoteFunction is now located in the decorators module

[fixes]

• Defaults for computer arguments can now be quoted with “ or ‘

• Fixed strange behaviour when dealing with strings addition in argument defaults

• Fixed an issue where DynamicValue would ignore the value property

[clean]

• Reduced argument objects no longer have redundant brackets surrounding a calculation

[tests]

• Testing suite does a better job of cleaning up after itself

• Fixed argument default tests that were testing the wrong thing

• Legacy computer unpack testing now actually uses the legacy method

0.11.5

[refactor]

• Remove unserialisation protections from SendableMixin. They were too easy to bypass, and causing bugs

• Runners now intialise and connect to the database at init

[fixes]

• Resource objects no longer drop their tag and separator attributes after a serial loop

• Fixed an instance where CMD.latency would over estimate the delay

• landing_dir no longer defaults to local $HOME

[docs]

• Updated installation info

• Added warning to intro regarding running with untrusted data

• Added the move_docs.py script, allowing for moving documentation pages while maintaining links

• Rearrange documentation layout

• Fix broken repository links

• Overhaul Computer documentation

• Minor clarity improvements

[clean]

• Added docs installation optional

• Update dev installation optional

• Several cleanup passes on various source files

[tests]

• add seveal tests to increase coverage

[CI]

• PyPi push is no longer allowed to fail

0.11.4

[features]

• BaseComputer can now accept a template, extracting and creating Substitution objects for any #VALUE# strings

• Substitution can now accept format

• Source code for a parser is now available at the parser_source property

[fixes]

• temporary runtime args no longer update only Resource objects

• fixed an issue where format_time would not bother to convert the input

[refactor]

• format is now stored in the DynamicMixin

0.11.3

[refactor]

• Substitution.entrypoint renamed to name

[features]

• Default args can now be quoted to prevent evaluation

0.11.2

[docs]

• Add a section to Dependency tutorial describing the nuances of environment variables

[feature]

• Added Substitution objects, a companion to Resource that replaces a target string with a variable

• Added arguments, argument_objects and argument_dict properties to BaseComputer

• Added substitutions, substitution_objects and substitution_dict properties to BaseComputer

• Resource objects can now specify their tag, defaults to --

• Resource objects can now specify their separator, defaults to =

• Added DynamicValue.static, which returns True if the value has no chain

• format_iterable can now also collect the object type with print_type=True

[refactor]

• BaseComputer resources, resource_objects and resource_dict return only Resource derived objects

• DynamicValues are now based on a DynamicMixin class

• Argument objects no longer rely on SendableMixin, using their reduced property

• Resource objects now use __slots__

• URL.python and URL.shebang are no longer property based attributes

• Transport no longer stores a copy of the URL, relying on Dataset to ensure it is properly populated

• Added Dataset.prepare_for_transfer(), which updates the Transport URL. Called by avoid_runtime()

• extra insertion is deferred entirely to the URL, allowing substitutions to work on temporary args

• Submission section can now be moved in the jobscript by adding "#SUBMISSION_SUBSTITUTION#"

• Toplevel imports are no longer relative

[fixes]

• Fixed an issue where Resource objects could be set improperly

• BaseComputer should now properly serialise connection parameters

• run_args in run() should no longer permanently update a BaseComputer

• BaseComputer.extra is now properly serialised

• Overwriting a DynamicValue that has a chain now properly prints a warning

• URL.port is no longer permanent

0.11.1

[fixes]

• Fixed an issue where a parser could be ignored by to_dict

[perf]

• Tweaked the method of Resource collection in to_dict

0.11.0

[docs]

• Add FAQ info on potential rsync workarounds

• Add FAQ info on updating a Computer to 0.11.x

• Overhaul Computer tutorials

[refactor]

• Overhaul Computer definitions

  • optional and required objects are removed, and replaced with a Resource object. Use optional=False/True args to enforce either

  • Replaced required_or spec field with requires and replaces keywords of Resource

  • Replaced optional_defaults spec field with default keyword of Resource

  • Resource outputs are cast to int if possible (or not blocked by format keyword)

  • parser can now be defined directly on the class, in a pythonic way

  • Added a limited ability for custom BaseComputer derived classes to define user customised functionality

  • BaseComputer now defines a default parser which should work reasonably well, or serve as a basis for your own

• Example_Slurm and Example_Torque no longer have an underscore in their names (ExampleSlurm & ExampleTorque)

• format_time function is now available at toplevel remotemanager.connection.computers

• Function will no longer remove self if it is found. (Can be forced added with force_self=True)

[features]

• Added DynamicValue, an internal value store for Resource that allows for deferred calculation. This enables setting dependent variables. For example: url.nodes = url.mpi * url.omp / url.cores_per_node

• parser now has access to self within the function

• Resource has a format keyword that accepts the “float” and “time” options

  • “float” allows output to be formatted as a float

  • “time” will convert integer seconds to HH:MM:SS format

• run_args are available within the parser at Resources["run_args"]

• Added resource_tag to BaseComputer and tag to Resource. This allows setting of the pre --flag “tag”

• Added resource_separator to BaseComputer and separator to Resource. This allows setting of the inter flag=value “separator”

[CI]

• Increase repeatability of some tests

v0.10.X

0.10.19

[fixes]

• CMD now warns if the stderr is empty, but returncode is nonzero

• Fixed an issue where CMD._file_communicate would be too agressive when reading files

[refactor]

• CMD now internally tracks the returncode

• Dataset.results and errors now uses avoid_runtime, prevents some race conditions

[docs]

• Add basic LaTeX config

0.10.18

[features]

• Dataset tries to replace any missing runner files on fetch_results()

• Added Dataset.retry_failed, reruns only runners marked with is_failed = True

• Added force_ignores_success as an argument for run(). Needs to be True for force to run Runners that are marked as is_success = True

• Added TrackedFile.exists_local property

[refactor]

• Runners will now not run if is_success (use force AND force_ignores_success to run)

• Transfer CMDs are available at Transfer.cmds

[docs]

• Add FAQ section on partially failed datasets

[performance]

• Minor optimisations on Transfers

0.10.17

[refactor]

• Added a shebang argument to URL. This is also accessible at the Dataset level.

0.10.16

[features]

• Runners now check that the remote python version is at least 3.x.x

[fixes]

• Function now ignores any content before def ..., allowing for decorators and comments

0.10.15

[features]

• Added BaseComputer.generate_cell() which will produce a string which can be copied into a Jupyter cell, producing an editable reproduction of the current Computer

• Added -q flag to URL.ssh by default, this should reduce the number of issues caused when messages are broadcast on stderr

• Dataset run_args are now available in BaseComputer.parser functions at resources["run_args"]

[fixes]

• Fixed an issue where functions containing @ characters would become mangled if stored as a Function

0.10.14

[features]

• BaseComputer.to_yaml() now prints the Parser in a more legible formats

• Added base.time_to_s, converting a HH:MM:SS string to integer seconds

[docs]

• Tweaked some docstrings and signatures

0.10.13

[fixes]

• BaseComputer.from_repo() now functions as expected

[refactor]

• Updating Dataset.dbfile is now more streamlined

[clean]

• clean instances of " f" autoformatting

• apply a pass of pycharm inspections

0.10.12

[docs]

• add FAQ

[feature]

• BaseComputer now dumps parser as a direct source string, rather than a Function object

[fixes]

• SanzuFunction can now handle functions with type hinting

0.10.11

[docs]

• Added missing complex serialisation tutorial

[fixes]

• Fixed a serialisation bug related to numpy (issue #8)

• Fixed an edge case where an object would pass json serialisation but not yaml

0.10.10

[features]

• append_run and insert_runner now have the option return_runner, which returns the appended runner

• Function now exposes a list of args at args

• SanzuFunction can now be called using non keyword args

[fixes]

• Fixed a bug where Function was unable to regenerate a python object.

  • This also fixes a related bug with BaseComputer

0.10.9

[features]

• Added SanzuFunction, allowing tagging of functions to run remotely

[refactor]

• Jupyter module renamed to Decorators

[docs]

• Extra Functions renamed internally to Decorators

0.10.8

[features]

• Dataset.wait now has the option success_only, which ignores failed runs

• Dataset now has a default_url property, allowing it to be set for all created Datasets

[fixes]

• Dataset.all_success now polls remote as expected

• sanzu function is now named __sanzu_fn, fixing an ambiguous name clash

[clean]

• remove Numpy dependency, only required for tests

• add functionality to clean documentation of pytest-nbval tags at docs/source/clean_docs

[docs]

• now cleaned of all pytest-nbval tags

[CI]

• Add retries to relevant tests

0.10.7

[docs]

• Update existing sanzu docs

• Add note on spull to sanzu docs

• Add note on exceptions to sanzu docs

[features]

• sanzu now returns the cell output

• sanzu will emit any warnings or errors it encounters

[refactor]

• sanzu now raises an exception if an empty cell is passed

0.10.6

[fixes]

• Changing the URL associated with a Dataset now also changes the corresponding Transport URL

• large run summary now correctly prints forced runs

• fixed an issue which prevented a url from being serialised after test_connection was issued

0.10.5

[features]

• Runners now have the set_run_arg suite of functions

• Dataset now has an all_success property, returning True if all runners have completed successfully

[fixes]

• RunnerStates no longer report as failed for incomplete runs

• Function no longer mangles signatures with renamed *args and **kwargs

[docs]

• Add a section on run_args

0.10.4

[features]

• ds.backup now has a full option, which will also collect extra_files_send

[fixes]

• url.passfile and keyfile are now settable properties

• Dataset.restore no longer overwrites an existing Dataset by default

0.10.3

[docs]

• Tweaked intro page on requirements

[refactor]

• Added a workaround for ast.unparse that allows the local machine to run python 3.7

0.10.2

[docs]

• Added Version History page

• Update index page

• Update note on version limtations

[features]

• Added sshpass_override, allowing overriding of the sshpass string

• URL now splits a "user@host" string

[fixes]

• Remove usages of | syntax from signatures

• Bump minimum python version to 3.9

[CI]

• Minimum python version (3.9) is now tested within the CI

0.10.1

[docs]

• Minor improvements to Dataset tutorial

[features]

• Added url.ssh_insert, allowing a custom string to be placed just after any ssh call

[refactor]

• BaseComputer.argument_dict is now much more clear

• host is now the first kwarg of URL

0.10.0

[fixes]

• Fixed an instance where a runner which raises a warning but ultimately completes would never be marked as such

[refactor]

• Runners now inherit their base run_args from the parent

• Dataset.run_args must now be set by the set_run_arg family of functions

[fixes]

• Runner run_args are now always valid for the current state

v0.9.X

0.9.24

[docs]

• Update introduction package overview image

• Added a section to intro on requirements

[fixes]

• Fixed an issue where a non-explicit remote directory would break result recovery

[features]

• Added url.expandvars(string) which will pass the string through the remote, expanding any variables

[refactor]

• Remote, local and run dirs are now sanitised to PurePosixPath

0.9.23

[features]

• Added boolean results, errors and extras args to fetch_results()

• Added ds.fetch_errors() which only fetches error files

[fixes]

• wait() now properly awaits single runs

• lazy_append context no longer errors if nothing is appended

• Fixed an issue where a directory sent with extra files could not be wiped

• Fixed an issue where sending a file named “*” could cause dangerous interactions with file wipes

[refactor]

• Dataset(skip=...) is no longer solely dependent on database presence

• Runners that have not been run now return None for their is_finished

0.9.21

[docs]

• API documentation is now much cleaner, modules are only documented within their module page

• Added dedicated section on complex serialisation

• Added a note on run_cmd to failure tutorial

• Made Quickstart quicker, and more relevant

[fixes]

• Fixed some instances where Function would improperly parse the signature

[clean]

• Removed some more instances of __all__ reliant imports

0.9.22

[refactor]

• Runners that fail now have a RunnerFailedError as their result

[fixes]

• Newly created runners no longer report as is_failed

0.9.20

[feature]

• Added ds.insert_runner(runner), allowing insertion of an unmodifed runner to a Dataset

• Added ds.copy_runners(dataset), copying runners from dataset into ds

[fixes]

• Fixed an instance where a Transport could hold a its own URL, separate to Dataset

[refactor]

• Runners will no longer run if their error file already exists (usually populated by a parent)

• Child jobs that have a pre-populated error file will pass it along the chain

• Runners now clear their result and error properties at run (files remain untouched)

[tests]

• Increased reproducibility of some unstable tests

[docs]

• Overhaul tutorial on handling failures

0.9.19

[features]

• Added the dir_mode option to Transport. On a multi-file transfer, files are copied into a temporary dir and copied using “*”, instead of using bash brace expansion

• Added URL.landing_dir, allowing the user to set the “landing” directory that commands are executed in by default. Functions by prepending a cd {landing_dir} && to each cmd

[fixes]

• Fixed an issue where a Dataset could repopulate its database incorrectly

[docs]

• Add docs sections for new features

0.9.18

[fixes]

• scp no longer tries to create the local target dir on the remote machine on pull

[refactor]

• scp now formats commands similarly to rsync

0.9.17

[fixes]

• Error files are no longer ignored if there is a result present

• Single runner runs no longer wipe all error files

[docs]

• Fix a dependency issue preventing docs compilation

0.9.16

[docs]

• Add documentation for the new Backup and Restore systems

• Trim Dataset tutorial

[features]

• Added Dataset.backup and Dataset.restore, allowing backing up, of a dataset and its results to a zip file for later recovery

• Added Database.backup, which will backup the current database file to file. Appends .bk to the current name by default

[fixes]

• Dataset not longer contaminates the extra_files_recv with send when doing a remote searches

• Fixed an issue where objects within lists would not be deserialised

0.9.15

[fixes]

• Runners that have not been run can no longer be considered for completion

• Dataset extra can now be set after initialisation

[refactor]

• BaseComputer is now importable from remotemanager

• SendableMixin now uses the same method for objects that use __slots__ and __dict__

[tests]

• Fix wrongly named test

0.9.14

[features]

• Added runner.full_error, returning the full content of the error file as a string

[fixes]

• Using a function called dump or load no longer silently fails

• ds.results now checks if there are errors and sends a warning if there are

• TrackedFile.content now returns None if the file does not exist

0.9.13

[fixes]

• Appending a runner that already exists now warns using the existing runner name

• Database no longer sorts runners alphabetically

• Database.find() now returns an empty dict if the value was not found

• Fixed an issue where an old Dataset could be updated into a new one

• Fixed an issue where sanzu would raise an error after a skipped run

• Update sed command to perform identically on *nix and BSD based OSes (@wddawson)

0.9.12

[features]

• avoid_nodes is now a run_arg, meaning you can use it as part of a dependency

[refactor]

• Improve TrackedFile.write, now attempts to convert args to string type

• TrackedFile.write now has a add_newline arg (defaults True)

• Moved the base Transport._cmd method into the cmd method

[clean]

• Replace instances of explicit TrackedFile writing with TrackedFile.write

0.9.11

[features]

• Lazy appends now only print a summary of the process after completion, rather than for each runner

• If the number of runners is more than Dataset.run_summary_limit (defaults to 25, settable), run() will print a summary instead of a per-runner update

0.9.10

[refactor]

• Runner now wipes and reinitialises any TrackedFile instances at run, making them “static”

• TrackedFile no longer has Logging access, allows full conversion to __slots__

• extra files are now handled via TrackedFile

• is_finished is now checked against the Runner files’ last_seen_remote

0.9.9

[fixes]

• SendableMixin now correctly handles objects that use __slots__

[refactor]

• check_runner_outputs now also checks for extra_files_recv

• Replaced an instance of old run path generation with runner.run_path

0.9.8

[refactor]

• Removes wildcard import from toplevel __init__

[fixes]

• Database.find() now searches more thoroughly. Use greedy=True for old behaviour

[clean]

• Update pre-commit hooks

0.9.7

[fixes]

• Master dir location passing is now done via a sed update of the jobscripts

• wipe_local no longer deletes local files from extra_files_send

0.9.6

[refactor]

• wipe_local now uses the same file collection methodology as wipe_remote

• hard_reset, wipe_local and wipe_remote now default to file_only=True

0.9.5

[features]

• Added runner.is_failed and runner.is_success, returning True/False if the runner has completed successfully or not

• Added runner.extra, which allows extra lines to be inserted into the jobscript of a runner. Use just like a run_arg, lines are inserted prior to submission.

[docs]

• Add a section regarding handling of failed runners

[CI]

• Tests are now split into sections, forcing some level of parallelisation

0.9.4

[fixes]

• Dataset master dir is now given to the runners via command line instead of environment vars

[features]

• Added success property of RunnerState, allows for extra context

• Added ds.failed, a list of runners reporting to have failed

0.9.3

[features]

• Added runner.set_state(), which allows force setting of the state (see fixes)

[fixes]

• Fixed an instance where an older run could masquerade as the results of a more recent one

• Runner state update now blocks the change if it would duplicate the state

0.9.2

[features]

• Adds ds.set_run_args, allowing a quick way of setting a run arg for each runner in the dataset

[fixes]

• Satisfied runners no longer re-update their status to “satisfied”

0.9.1

[fixes]

• Lazy append is now O(N) when appending to a dataset that has a linked dependency

0.9.0

[refactor]

• Drastically changes how runners are handled within a run

• TrackedFiles are now much more prevalent, and have increased power

• Dependencies can now be run from anywhere in the chain

[features]

• Runner states are now handled in a RunnerState class, allowing for more precise state checking and tracking

• TrackedFile allows for obtaining a relative_remote_dir(source), which gives a callpath from source to the file

• TrackedFile now has a write method

• Transport now has direct support for a TrackedFile

• Added utils.dir_delta, which gives the relative directory level difference between two directories